Re-TACRED: Addressing Shortcomings of the TACRED Dataset

TACRED is one of the largest and most widely used sentence-level relation extraction datasets. Proposed models that are evaluated using this dataset consistently set new state-of-the-art performance. However, they still exhibit large error rates despite leveraging external knowledge unsupervised pretraining on text corpora. A recent study suggested may be due to poor quality. The observed over 50% challenging sentences from development test sets incorrectly labeled account for an average drop 8% f1-score in model was limited a small biased sample 5k (out total 106k) sentences, substantially restricting generalizability broader implications its findings. In paper, we address these shortcomings by: (i) performing comprehensive whole dataset, (ii) proposing improved crowdsourcing strategy deploying it re-annotate (iii) thorough analysis understand how correcting annotations affects previously published results. After verification, 23.9% labels incorrect. Moreover, evaluating several our revised yields improvement 14.3% helps uncover significant relationships between different (rather than simply offsetting or scaling their scores by constant factor). Finally, aside also release Re-TACRED, completely re-annotated version can perform reliable evaluation models.